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Program description 



Progress in Mathematics © 2006 is a new core curriculum for 
students in kindergarten through grade 6. Progress in Mathe- 
matics © 2006 differs substantively from Progress in Mathemat- 
ics © 2000 in both content and assessment material. Progress 
in Mathematics © 2006 uses a sequence of systematic lesson 
plans to teach mathematical concepts and skills. It incorporates 
the following features at each grade level: explicit instruction of 
mathematics content; development of conceptual understanding 



through a three-step process that begins with hands-on activi- 
ties (concrete thinking to visual thinking to symbol use); fluency 
in numerical computation; problem solving; development of 
mathematical vocabulary; practice and review; and different 
types of assessment. Student textbooks, student workbooks, 
and teacher’s editions are available for each grade level, as well 
as manipulatives and online practice exercises. 



R6S6arch One study of Progress in Mathematics © 2006 met the What The WWC considers the extent of evidence for Progress in 

Works Clearinghouse (WWC) evidence standards. The study Mathematics © 2006 to be small for math achievement, 
included 186 first grade students in eight classrooms across four 
schools located in New York and Pennsylvania.'' 



Eff6CtiV6n6SS Progress in Mathematics © 2006 was found to have no discernible effects on math achievement. 

Math achievement 
Rating of effectiveness No discernible effects 
Improvement index^ Average: +3 percentile points 

Range: -17 to +22 percentile points 



1. The evidence presented in this report is based on avaiiabie research. Findings and conciusions may change as new research becomes avaiiable. 

2. These numbers show the improvement index for the one finding in the study used for the rating. 
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Developer and contact 

Developed and distributed by Sadlier-Oxford, a division of 
William H. Sadiier, Inc. Address: William H. Sadiier, Inc., Sadlier- 
Oxford Division, 9 Pine Street, New York, NY 10005. Email: 
CustomerService@sadlier.com. Web: www.sadiier-oxford. 
com/math/index.cfm. Telephone: (800) 221-5175. 

Scope of use 

The original Progress in Mathematics curriculum was developed 
in the 1940s by the Sisters, Servants of the Immaculate Heart 
of Mary. William H. Sadiier, Inc. has published the program for 
more than 60 years. Earlier versions of the program have been 
used in many Catholic elementary schools across the country, 
but specific information is not available on the number or 
demographics of students, schools, or districts using Progress 
in Mathematics © 2006? 

Teaching 

As part of the Progress in Mathematics © 2006 curriculum, 
students work with a textbook and a companion workbook to 
learn a variety of math content, including number and arithmetic 
operations, pre-algebraic thinking, geometry, data and statistics, 
logic, and estimation. Each lesson ot Progress in Mathematics 



© 2006 features Math Words, identifying new math vocabulary 
words for the lesson; a mixture of computational exercises; a 
“Talk It Cver” feature designed to help children learn to sum- 
marize the teaching and communicate it mathematically; and a 
special feature, such as Mental Math (doing calculations without 
physical aids such as paper and pencil), at the end of the lesson. 

Teachers receive a pre-implementation orientation and ongo- 
ing instructional and technical support in person or by phone 
from the developer. In addition, an interactive website is available 
for students and teachers, which includes further practice and 
enrichment and teaching tips. 

Cost 

A two-volume set of student textbooks for Progress in Math- 
ematics © 2006 costs $30.69; a single-volume student textbook 
costs $28.95. The e-book version of the student textbook costs 
$28.95. The teaching manual costs $98.85. Additional materials 
include a student workbook ($7.98), a teacher’s edition of the 
workbook ($12.00), a student test booklet ($42.00), a teacher’s 
edition of the student test booklet ($12.00), an additional 
practice/test generator CD-ROM ($147.00), and a manipulatives 
package ($21.00). 



One study reviewed by the WWC investigated the effects of 
Progress in Mathematics © 2006. This study (Beck Evaluation & 
Testing Associates, 2005) was a randomized controlled trial that 
met WWC evidence standards. 

The Beck Evaluation & Testing Associates (2005) study 
included 186 first-grade students in eight classrooms across 
four schools. Three schools were located in New York, and one 
school in Pennsylvania. Each school identified two first grade 
classrooms for the study: one classroom was randomly assigned 
to the intervention group and the other assigned to the compari- 



son group. Thus there were a total of four classrooms in the inter- 
vention group and four classrooms in the comparison group. The 
intervention classrooms used a pre-publication comparison of 
the Progress in Mathematics © 2006 program. The comparison 
classrooms used the earlier and substantively different © 2000 
version of Progress in Mathematics. 

Extent of Evidence 

The WWC categorizes the extent of evidence in each domain as 
small or moderate to large (see the What Works Clearinghouse 



3. This intervention report regards Progress in Mathematics © 2006 as a different program from Progress in Mathematics © 2000. The WWC team com- 
pared the text books of both programs and found them to differ extensively in terms of content, assessment materials, organization, and presentation. 
Information received from the developer confirmed this difference between programs. 
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Effectiveness 



The WWC found Progress 
in Mathematics © 2006 to 
have no discernible effects 
for math achievement 
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Extent of Evidence Categorization Schemed . The extent of The WWC considers the extent of evidence for Progress in 

evidence takes into account the number of studies and the Mathematics © 2006 to be small for math achievement, 

total sample size across the studies that met WWC evidence 
standards with or without reservations.'* 



Findings 

The WWC review of interventions for elementary school math 
addresses student outcomes in one domain: mathematics 
achievement. 

The Beck Evaluation & Testing Associates (2005) study 
reported a statistically significant positive effect of the Progress 
in Mathematics © 2006 program on the Terra Nova Mathematics 
Test (referred to in the study as Part 1); however, this effect was 
not statistically significant according to WWC analysis. The study 
reported no statistically significant effect on the Terra Nova Math- 
ematics Computation Test (referred to in the study as Part 2). The 
average effect size across the two student outcomes was neither 
statistically significant nor large enough to be considered substan- 
tively Important according to WWC standards (at least 0.25). 



In sum, one study of Progress in Mathematics © 2006 found 
no discernible effects on students’ math achievement. 

Rating of effectiveness 

The WWC rates the effects of an intervention in a given outcome 
domain as: positive, potentially positive, mixed, no discernible 
effects, potentially negative, or negative. The rating of effective- 
ness takes into account four factors: the quality of the research 
design, the statistical significance of the findings,® the size of 
the difference between participants in the intervention and the 
comparison conditions, and the consistency in findings across 
studies (see the WWC Intervention Rating Scheme) . The WWC 
found Progress in Mathematics © 2006 to have no discernible 
effects for math achievement. 



Improvement index 

The WWC computes an improvement index for each individual 
finding. In addition, within each outcome domain, the WWC 
computes an average improvement index for each study and an 
average improvement index across studies (see Technical Details 
of WWC-Conducted Computations) . The improvement index rep- 
resents the difference between the percentile rank of the average 
student in the intervention condition versus the percentile rank 
of the average student in the comparison condition. Unlike the 
rating of effectiveness, the improvement index is based entirely 
on the size of the effect, regardless of the statistical significance 
of the effect, the study design, or the analyses. The improvement 



index can take on values between -50 and +50, with positive 
numbers denoting results favorable to the intervention group. 

The average improvement index for mathematics achievement 
is +3 percentile points with a range of -17 to +22 percentile 
points in the single study reviewed. 

Summary 

The WWC reviewed one study on Progress in Mathematics © 
2006. This single study met WWC evidence standards. Based 
on this study, the WWC found no discernible effects in the math 
achievement domain. The evidence presented in this report is 
limited and may change as new research emerges. 



4. The Extent of Evidence Categorization was developed to tell readers how much evidence was used to determine the intervention rating, focusing on the 
number and size of studies. Additional factors associated with a related concept, external validity, such as students’ demographics and the types of 
settings in which studies took place, are not taken into account for the categorization. 

5. The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within 
ciassrooms or schools and for multiple comparisons. For an explanation, see the WWC Tutorial on Mismatch . See Technical Details of WWC-Conducted 
Computations for the formulas the VVWC used to calculate the statistical significance. In the case of Progress in Mathematics © 2006, corrections for 
ciustering and multiple comparisons were needed. 
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References 


Met WWC evidence standards 

Beck Evaluation & Testing Associates, Inc. (2005). Progress in 
Mathematics © 2006: Grade 1 pre-post field test evaluation 
study. New York: Sadlier-Oxford Division, William H. Sadlier, 
Inc. 




For more information about soecific studies and WWC caicuiations. niease see the WWC Proaress in 
Mathematics © 2006 Technicai AoDendices. 
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Appendix 



Appendix A1 


Study characteristics: Beck Evaiuation & Testing Associates, 2005 (randomized controiied triai) 


Characteristic 


Description 


Study citation 


Beck Evaluation & Testing Associates, Inc. (2005). Progress in Mathematics © 2006: Grade 1 pre-post field test evaiuation study. New York: Sadlier-Oxtord Division, William H. 
Sadiier, Inc. 


Participants 


The study included 186 first graders (96 students in the intervention group and 90 students in the comparison group) in eight classrooms across four schools. Within schools, 
classrooms were randomly assigned to the intervention or comparison group. For rating purposes, the sample for the analysis of the Terra Nova Mathematics Test included 
186 students, and the sample for the analysis of the Terra Nova Math Computation Test included 181 students. 


Setting 


The eight classrooms were located in four elementary schools in four school districts in the eastern United States. Three of the schools were Catholic schools, and one was a 
public school. One pair of classrooms (one intervention and one comparison) was located in each of the participating schools. 


Intervention 


The intervention classrooms received the pre-publication version of Progress in Mathematics © 2006 student edition materials, student workbooks, and teacher guides. The 
study indicated that those materials resembled as closely as possible the intended published version. 


Comparison 


The comparison classrooms used the 2000 version of Progress in Mathematics. This textbook series had been used in the participating schools for at least three years prior to 
the study. This intervention report regards Progress in Mathematics © 2006 as a different math program from Progress in Mathematics © 2000. The WWC team compared 
the textbooks of both programs and found them to differ extensively in terms of content, assessment materials, organization, and presentation. Whereas the 2000 version 
emphasizes written computation skills, the © 2006 version focuses on mathematical language and problem solving in addition to computation. Information received from the 
developer confirmed this difference between programs. 


Primary outcomes 
and measurement 


Students were tested using the TerraNova Mathematics and Math Computation Tests (see Appendix A2 for more detailed descriptions of outcome measures).^ 


Teacher training 


Intervention group teachers received a pre-implementation orientation from the developer’s editorial staff. They also received ongoing editorial department support through 
in-person visits and by phone throughout the study.^ The comparison group teachers already had previous training and experience with their current textbooks. 



1 . The study reported on student outcomes using an additional outcome measure, the Custom Test, which did not meet WWC evidence screens because of differential attrition of students in the 
intervention and comparison groups. 

2. This study does not provide information about whether this level of training and ongoing support is reflective of the program’s typical implementation. 
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Appendix A2 Outcome measures in the math achievement domain 



Outcome measure 


Description 


TerraNova Mathematics Test 


Level 11, Form C of the CAT TerraNova series, seoond edition (MoGraw-Hill, 2001; as cited in Beck Evaiuation and Testing Associates, 2005) is a standardized nationally 
normed test. This mathematics test inciudes 47 items and is part of the CAT Basic Battery. 


TerraNova Mathematics 
Computation Test 


Level 11, Form C of the CAT TerraNova series, second edition (McGraw-Hill, 2001; as cited in Beck Evaiuation and Testing Associates, 2005) is a standardized nationally 
normed test. This test includes 20 mathematics computation items and is part ot the CAT Pius portion ot the Terra Nova series. 
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Appendix A3 Summary of study findings inciuded in the rating for the math achievement domain^ 









Authors’ findings from the study 














Mean outcome 
(standard deviation^) 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size 
(classrooms/ 
students) 


Progress in 

Mathematics Comparison 

groups group 


Mean difference'* 
{Progress in 
Mathematics - 
comparison) 


Statistical 
significance^ 
Effect size® (at a = 0.05) 


Improvement 

index^ 



Beck Evaluation & Testing Associates, 2005 (randomized controlled trial)^ 



TerraNova Mathematics Test 


Grade 1 


8/186 


40.62 

(4.30) 


37.70 

(5.80) 


2.92 


0.57 


ns 


+22 


TerraNova Mathematics 
Computation Test 


Grade 1 


8/181 


15.50 

(2.70) 


16.80 

(3.30) 


-1.30 


-0.43 


ns 


-17 



Domain average^ for math achievement (Beck Evaluation & Testing Associates, 2005) 0.07 ns +3 



ns = not statistically significant 

1 . This appendix reports findings considered for the effectiveness rating and the average improvement indices. Subtest findings from the same study are not included in these ratings, but are reported in Appendix A4. 

2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes. 

3. The intervention group mean equals the comparison group mean plus the mean difference. 

4. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. The mean difference and effect size were calculated using the difference in difference approach, 
which takes baseline student scores into account. 

5. For an explanation of the effect size calculation, see lechnical Details of WWC-Conducted Computations . 

6. Statistical significance is the probability that the difference between the groups is a result of chance rather than a real difference between the groups. 

7. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. The improvement index can take on values 
between -50 and +50, with positive numbers denoting results favorable to the intervention group. 

8. The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple comparisons. For an explanation about the clus- 
tering correction, see the WWC Tutorial on Mismatch . See Technical Details of WWC-Conducted Computations for the formulas the WWC used to calculate statistical significance. In the case of Beck Evaluations & Testing Associates 
(2005), corrections for clustering and multiple comparisons were needed, so the significance levels differ from those reported in the original study. 

9. This row provides the study average, which in this instance is also the domain average. The WWC-computed domain average effect size is a simple average rounded to two decimal places. The domain improvement index is calculated 
from the average effect size. 
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Appendix A4 Progress in Mathematics © 2006 rating for the math achievement domain 



The WWC rates an intervention’s effects in a given outcome domain as positive, potentially positive, mixed, no discernible effects, potentially negative, or negative.^ 

For the outcome domain of math achievement, the WWC rated Progress in Mathematics © 2006 as having no discernible effects. It did not meet the criteria for other 
ratings (positive effects, potentiaiiy positive effects, mixed effects, potentiaily negative effects, and negative effects) because the single study that met WWC standards 
did not show statisticaiiy significant or substantively important effects. 

Rating received 

No discernible effects: No affirmative evidence of effects. 

• Criterion 1: None of the studies shows a statisticaiiy significant or substantively important effect, either positive or negative. 

Met. The singie study that assessed outcomes in this domain showed indeterminate effects. 

Other ratings considered 

Positive effects: Strong evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: Two or more studies showing statisticaiiy significant positive effects, at ieast one of which met WWC evidence standards for a strong design. 

Not met. No studies showed statistically significant or substantiveiy important positive effects. 

• Criterion 2: No studies showing statisticaiiy significant or substantiveiy important negative effects. 

Met. No studies showed statistically significant or substantively important negative effects. 

Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: At least one study showing a statisticaiiy significant or substantively important positive effect. 

Not met. No studies showed a statisticaiiy significant or substantively important positive effect. 

• Criterion 2: No studies showing a statisticaiiy significant or substantiveiy important negative effect and fewer or the same number of studies showing indeterminate 
effects than showing statistically significant or substantiveiy important positive effects. 

Not met. The singie study on Progress in Mathematics © 2006 showed indeterminate effects. 

Mixed effects: Evidence of inconsistent effects as demonstrated through either of the foiiowing criteria. 

• Criterion 1: At least one study showing a statisticaiiy significant or substantively important positive effect, and at least one study showing a statisticaiiy significant 
or substantively important negative effect, but no more such studies than the number showing a statistically significant or substantively important positive effect. 

Not met. No studies showed a statisticaiiy significant or substantively important effect. 

• Criterion 2: At ieast one study showing a statistically significant or substantively important effect, and more studies showing an indeterminate effect than showing 
a statisticaiiy significant or substantiveiy important effect. 

Not met. No studies showed a statisticaiiy significant or substantively important effect. 



(continued) 
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Appendix A4 Progress in Mathematics © 2006 rating for the math achievement domain (continued) 



Potentially negative effects: Evidence of a negative effect with no overriding contrary evidence 

• Criterion 1: At least one study showing a statisticaiiy significant or substantively important negative effect. 

Not met. No studies showed a statisticaiiy significant or substantively important negative effect. 

• Criterion 2: No studies showing a statistically significant or substantively important positive effect, or more studies showing statistically significant or substantively 
important negative effects than showing statistically significant or substantively important positive effects. 

Met. No studies showed a statistically significant or substantively important positive effect. 

Negative effects: Strong evidence of a negative effect with no overriding contrary evidence. 

• Criterion 1: Two or more studies showing statistically significant negative effects, at least one of which met WWC evidence standards for a strong design. 

Not met. No studies showed a statistically significant negative effect. 

• Criterion 2: No studies showing statistically significant or substantively important positive effects. 

Met. No studies showed a statistically significant or substantively important positive effect. 



1 . For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain-level effect. The WWC also considers the size of the domain-level effect for ratings of 
potentially positive or potentially negative effects. See the WWC Intervention Rating Scheme for a complete description. 
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Appendix A5 Extent of evidence by domain 



Outcome domain 


Number of studies 


Schools 


Sample size 


Students 


Extent of evidence^ 


Math achievement 


1 


4 




181 


Small 



1 . 



A rating of “moderate to large” requires at least two studies and two schools across studies in one domain, and a total sample size across studies of at least 350 students or 14 classrooms. 
Otherwise, the rating is “small.” 
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